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Abstract 

This  paper  examines  algorithms  for  detecting  when  a  property  $  holds 
during  the  execution  of  a  distributed  system.  The  properties  we  con¬ 
sider  are  expressed  over  the  state  of  the  system  and  are  not  aissumed 
to  have  properties  that  facilitate  detection,  such  as  stability. 

Detection  is  done  by  a  monitoring  process  within  the  system,  which 
cannot  perceive  an  execution  of  a  distributed  system  as  a  total  order: 
because  of  this,  we  consider  two  interpretations  for  "detecting  <^" : 

1.  There  is  an  execution  consistent  with  the  observed  behavior  such 
that  $  was  true  at  a  point  in  that  execution.  We  refer  to  this 
property  as  possibly 

2.  For  all  executions  consistent  with  the  observed  behavior,  there 
was  some  point  in  real  time  at  which  the  global  state  of  the 
system  satisfied  <5.  We  refer  to  this  property  as  definitely  $. 

In  this  paper,  we  give  formal  definitions  for  these  two  interpreta¬ 
tions  and  present  algorithms  for  them.  We  give  protocols  for  both 
asynchronous  and  synchronous  systems  and,  for  synchronous  systems, 
give  upper  bounds  on  the  time  between  the  occurrence  of  the  property 
of  interest  and  the  time  a  monitor  detects  the  property. 
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1  Introduction 

A  reactive  system  [6]  is  characterized  by  a  control  program  that  interacts 
with  an  environment.  The  control  program  is  input-driven:  it  monitors  the 
environment  and  reacts  to  significant  events  by  sending  commands  to  the 
environment.  There  are  many  examples  of  reactive  systems;  for  example, 
most  embedded  real-time  systems  are  reactive  systems,  in  which  case  the 
environment  is  an  instrumented  physical  process.  Non-real-time  examples 
of  reactive  systems  includes  monitoring  and  debugging  systems  [4.13]  and 
tool  integration  services  [5,14]. 

In  the  Meta  project  [9,11],  we  have  been  developing  tools  that  support 
the  management  of  distributed  applications  through  the  use  of  a  reactive 
system  structure.  Using  Meta,  the  distributed  application  and  its  supporting 
services  (for  example,  operating  system,  network  servers,  and  hardware)  can 
be  instrumented  with  sensors  that  access  its  state  and  actuators  that  allow 
its  state  to  be  changed.  Meta  also  provides  a  distributed  interpreter  of 
finite  state  automata  that  reference  these  sensors  and  actuators.  Under 
Meta,  control  programs  are  translated  into  finite  state  automata  that  are 
executed  by  this  distributed  interpreter.  Each  interpreter  executes  guarded 
atomic  commands  of  the  form  ($  — ►  S),  meaning  e.xecute  the  action  S  in  a 
state  satisfying  the  global  state  predicate  $. 

The  problem  addressed  in  this  paper  arises  in  the  context  of  Meta:  how 
can  a  set  of  processes  monitor  the  state  of  a  distributed  application  in  a 
consistent  manner?  For  example,  consider  the  simple  distributed  application 
shown  in  Figure  1.  Each  of  the  three  processes  in  the  application  has  a 
light,  and  the  control  processes  would  each  like  to  take  an  action  when 
some  specified  subset  of  the  lights  are  on.  The  application  processes  are 
instrumented  with  stubs  that  determine  when  the  process  turns  its  light  on 
or  off.  This  information  is  disseminated  to  the  control  processes,  each  of 
which  then  determines  when  its  condition  of  interest  is  met. 
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Meta  is  built  on  top  of  the  ISIS  toolkit  [1],  and  so  we  first  built  the 
sensor  dissemination  mechanism  using  atomic  broadcast,  .ktomic  broadcast 
guarantees  that  all  recipients  receive  the  messages  in  the  same  order  and 
that  this  order  is  consistent  with  causality  [7],  Unfortunately,  the  control 
processes  are  somewhat  limited  in  what  they  can  deduce  when  they  find 
that  their  condition  of  interest  holds. 

For  example,  Figure  2  shows  a  space-time  diagram  of  an  e.xecution  of 
the  application  shown  in  Figure  1.  In  this  figure,  a  process  turning  its  light 
on  is  represented  by  a  rectangle  and  the  process  turning  its  light  off  is  rep- 
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Figure  1:  A  Monitored  Distributed  Application 


resented  by  a  vertical  line.  Assume,  for  the  moment,  that  this  system  is 
asynchmnotis,  meaning  that  there  is  no  bound  on  message  passing  delays  or 
on  the  relative  speeds  of  processes.  In  this  case,  the  only  ordering  relations 
between  events  that  can  be  determined  from  within  the  system  are  those  of 
potential  causality.  Two  events  that  are  not  so  related  are  concurrent.  In 
Figure  2,  the  events  a  and  b  are  concurrent  as  are  a  and  c.  so  the  control  pro¬ 
cesses  could  receive  these  event  notifications  (as  sent  by  atomic  broadcast)  in 
one  of  these  orders:  (a;6;c),  {b;a:c)  or  (6:c;a).  Thus,  the  control  processes 
may  or  may  not  determine  that  pi’s  and  p2’s  lights  were  on  simultaneously, 
but  they  will  reach  the  same  decision.  On  the  other  hand,  the  events  a.  d 
and  e  are  causally  ordered,  so  the  control  processes  will  determine  that  pi's 
and  Pa’s  lights  were  on  simultaneously. 

Given  a  global  property  #,  there  are  at  least  two  ways  th.r,  "detecting 
can  be  interpreted: 

1.  There  is  an  execution  (i.e.  a  linear  sequence  of  events)  consistent 
with  the  observed  behavior  such  that  $  was  true  at  a  point  in  that 
execution.  We  will  refer  to  this  property  as  possibly  In  the  space- 
time  diagram  shown  in  Figure  2,  the  pre'.icate  imssibly  (pi's  light  on 
and  Pa’s  light  on)  holds. 
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Figure  2:  Space-Time  Diagram  of  Application  Execution 

2.  For  all  executions  consistent  with  the  observed  behavior,  there  was 
some  point  in  real  time  at  which  the  global  state  of  the  system  satisfied 
We  will  refer  to  this  property  as  definitely  In  the  execution 
shown  in  Figure  2,  the  predicate  definitely  (pi’s  light  on  and  p3‘s  light 
on)  holds,  since  the  event  of  p3  turning  its  light  on  happened  between 
Pi  turning  its  light  on  and  pi  turning  its  light  off. 

Note  that  definitely  $  is  stronger  than  possibly  Hence,  we  will  want 
to  guarantee  that  if  a  control  program  determines  possibly  ^  for  a  set  of 
local  states,  then  no  control  program  will  ever  determine  definitely  for 
the  same  states.  Note  that  both  of  these  conditions  refer  to  some  past  state 
or  states. 

In  this  paper,  we  give  formal  definitions  for  these  two  interpretations 
and  present  protocols  for  them.  We  first  give  the  protocols  for  an  asyn¬ 
chronous  system.  These  protocols  can  take  an  unbounded  amount  of  time 
to  detect  their  condition  of  interest;  furthermore,  they  can  have  substan¬ 
tial  running  times  because  they  may  have  to  enumerate  may  possible  global 
states.  However,  no  better  is  possible,  in  general,  due  to  the  nature  of 
asynchronous  systems.  We  then  modify  these  protocols  for  a  system  with 
approximately  synchronized  clocks  and  bounded  message  delay.  These  pro¬ 
tocols  are  more  practical,  and  we  give  upper  bounds  on  the  time  between 
the  occurrence  of  the  property  of  interest  and  the  time  a  control  program 
detects  the  property.  The  existence  of  such  a  bound  makes  these  protocols 
more  useful  in  real  systems. 

Snapshot  protocols  for  computing  global  states  of  a  distributed  system  [2] 
are  related  to  the  protocols  described  in  this  paper,  but  they  suffer  from  a 
limitation  similar  to  that  of  the  atomic  broadcast  implementation  described 
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above.  In  particular,  if  S  is  the  global  state  computed  by  the  snapshot, 
then  there  exists  a  legal  execution  of  the  system  containing  5.  and  so  $(5) 
implies  possibly  $.  Snapshot  protocols  are  well-suited  for  detecting  stable 
properties,  which  are  those  that,  once  they  become  true,  the  remain  so.  It 
may  be  the  case  that  possibly  $  holds  of  an  execution,  but  the  snapshot 
protocol  never  detects  it  (this  can  happen  if  $  is  not  stable). 

A  recent  dissertation  by  Spezialetti  [16]  looks  at  a  broader  set  of  issues 
than  those  covered  in  this  paper,  such  as  using  semantic  information  (like 
relative  stability)  to  determine  which  local  events  could  make  a  global  prop¬ 
erty  true.  This  dissertation  also  presents  protocols  whose  specification  is 
similar  to  ours.  However,  her  protocols  that  detects  event  occurrence  suf¬ 
fer  from  the  same  limitation  as  snapshots  and  the  atomic-broadcast-based 
protocol  described  above.  Additionally,  Spezialetti's  protocols  do  not  take 
into  account  the  ordering  of  events  established  by  the  underlying  system's 
communication.  We  have  also  looked  at  the  problem  addressed  in  this  paper 
when  environments  are  continuous  state  transition  systems  [8].  Such  sys¬ 
tems  have  the  useful  property  that  physical  variables  can.  in  many  cases,  be 
interpolated  forward.  By  doing  so,  the  monitor  can  reason  about  the  current 
state  of  the  physical  process  rather  than  a  past  state,  and  so  [wssibly  $  and 
definitely  $  can  be  determined  for  the  current  state. 

2  Definitions 

We  first  define  the  notion  of  an  execution  of  a  system.  system  is  composed 
of  processes,  some  of  which  are  part  of  the  application  being  run  and  some 

of  which  are  part  of  the  monitoring  control  program.  Let  {/)] . p„}  be 

the  set  of  application  processes;  for  the  sake  of  simplicity,  we  assume  that 
there  is  only  one  monitoring  process,  denoted  po-  Each  pair  of  processes  is 
connected  by  a  point-to-point,  reliable,  fifo  communication  link,  and  we 
assume  that  processes  do  not  exhibit  faulty  behavior. 

Each  process  p,  has  a  local  state  Si,  which  changes  when  an  event  occurs 
at  the  process.  An  event  may  be  completely  internal  to  the  process,  or  it 
may  be  the  sending  or  receipt  of  a  message  (e.g.,  "send  mi  to  p/'  or  "receive 
mj  from  p*”).  For  the  sake  of  simplicity,  we  assume  that  all  message  sent  in 
the  system  are  unique.  Process  p,'s  local  history,  denoted  h,  is  a  (possibly 
infinite)  sequence  of  states  and  events 


In  this  case,  s®  is  pi's  initial  state,  and  the  first  event  it  executes  is  e'. 
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after  which  the  process’s  state  is  sf,  etc.  A  global  state  is  a  tuple  S  = 
(so?'Si»  •  •  one  for  each  process.  Although  the  monitor,  po.  is  a  process 

in  the  system,  when  we  refer  to  a  global  state,  we  will  usually  mean  only  the 
global  state  of  the  application,  (si,...,s„}.  A  global  history  (or  history)  is 
a  tuple  H  =  {ho,  hi,. . hn)  of  local  histories,  one  per  process. 

Although  a  global  history  does  not  specify  the  relative  timings  of  events 
and  states  at  different  processes,  it  does  allow  us  to  draw  certain  conclusions 
about  these  timings.  An  event  e[  happens  before  ej*  (written  e{  —  e’J')  if 
one  of  the  following  is  true  [7]: 

•  the  events  are  at  the  same  process  and  occur  in  the  order  indicated, 
that  is,  if  i  =  j  and  /  <  m; 

•  e{  is  the  sending  of  a  message  by  p,  to  pj  and  ej*  is  the  receipt  of  that 
message;  or 

•  there  is  another  event  ej  such  that  e[  —  eJJ  and  eJJ  —  e'J'. 

The  “happens  before”  relation  can  be  used  to  reason  about  the  possible 
executions  associated  with  a  global  history.  We  associate  with  each  global 
history  a  set  of  linearizations. 

A  linearization  i  of  a  history  H  is  &  sequence  of  global  states  and  local 
events 

SV5‘e252--- 

that  contains  exactly  the  events  in  H  such  that,  if  e’^  —  e”  in  H .  then  m  < 
n.  Notice  that  no  prefix  of  L  contains  the  receipt  of  a  message  whose  sending 
does  not  appear  in  that  prefix.  In  synchronous  systems  (see  Section  3.3). 
there  are  further  constraints  on  the  linearizations  of  a  global  history. 

(The  above  definition  of  linearization  assumes  that,  in  the  actual  ex¬ 
ecution  of  a  distributed  system,  no  two  events  can  occur  simultaneously. 
This  need  not  be  the  case;  it  is  possible  that  events  at  different  processors 
may  occur  at  exactly  the  same  time.  We  can  easily  extend  our  definition  of 
linearization  to  include  such  definitions. ) 

We  use  the  notion  of  a  cut  to  represent  the  global  states  that  could  have 
occurred  in  the  execution.  A  cut  [2J  of  a  global  history  H  is  a  tuple  of 
natural  numbers  (fi, . .  .,tn)  that  represents  the  state  of  the  system  after  /, 
events  have  executed  at  process  p,;  that  is.  the  cut  represents  the  global  state 
{sj‘, . .  Only  certain  cuts  of  a  global  history  can  truly  correspond  to 

global  states  that  took  place  at  some  real  time.  .\  cut  {ti . („)  of  H  is 

consistent  if  there  is  some  point  in  some  linearization  L  of  H  by  which  each 
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process  pi  executed  exactly  ti  events.  L  is  said  to  pass  through  this  cut.  We 
will  also  refer  to  the  associated  tuple  of  events.  , . . . ,  ef,"^  as  a  consistent 

cut. 

We  want  to  be  able  to  reason  about  certain  facts  (such  as  "possibly 
$”)  being  true  in  different  global  histories.  To  this  end,  we  introduce  the 
following  notation.  Let  H  be  some  global  history  of  the  system.  To  formally 
define  ‘^possibly  we  introduce  the  formulas  ?$1C,  where  C  is  a  consistent 
cut.  Formally,  ?$(C  holds  for  history  H  if  C  =  is  a  consistent 

cut  of  H  and  $  holds  for  the  global  state  (ij*, . . ., If  ?$|C  holds  for  H. 
then  it  is  possible  that  $  held  during  the  execution  that  generated  H  since 
it  held  at  some  point  in  some  linearization  of  H. 

To  formally  define  “'definitely  we  introduce  the  formulas  !$1.4.  where 
A  is  a  finite  set  of  cuts.  Formally,  !$|.4  holds  for  H  if  .4  is  a  finite  set  of 
consistent  cuts  of  H,  every  linearization  of  H  passes  through  some  cut  in  .4. 

and  for  all  cuts  (ti, . .  .,?„)  €  A,  $  holds  for  the  global  state  . 

If  !$|A  holds  for  H,  then  $  definitely  held  at  some  point  in  the  e.xecution 
that  generated  H  because  it  held  at  some  point  in  all  linearizations  of  H . 

Note  that  the  definitions  of  these  formulas  satisfy  two  properties  dis¬ 
cussed  earlier.  The  definitely  operator  !  is  clearly  stronger  than  the  possibly 
operator  ?  in  the  following  sense:  if  !$|A  holds  for  H  then  for  any  C  €  .4. 
?$(C  holds  for  H .  Furthermore  the  two  operators  are.  in  a  certain  sense, 
dual.  If  !“>$|A  holds  for  H,  then  ?$|C  cannot  hold  for  H  \f  C  £  .4. 

Informally,  the  control  process  po  detects  possibly  $  when  it  can  deter¬ 
mine  that  there  is  a  consistent  cut  of  H  that  satisfies  $.  and  po  detects 
definitely  $  when  it  can  find  a  finite  set  of  consistent  cuts  A  such  that  every 
linearization  of  H  passes  through  a  member  of  .4  and  such  that  $  holds 
for  every  member  of  A.  We  are  investigating  the  more  formal  definition  of 
detection,  but  we  do  not  present  such  defintions  in  this  version  of  the  paper. 

3  Protocols 

As  noted  above,  system  consists  of  n-f- 1  processes  {poi  P\ . PnJ  whose  only 

method  of  interaction  is  by  exchanging  messages.  The  process  po  monitors 
the  other  processes  to  determine  when  some  state  predicate  becomes  true. 
This  state  predicate  of  interest  will  be  of  the  form  possibly  $  or  definitely 
$,  where  $  is  a  predicate  over  the  states  of  the  processes  pi, - p^. 

Each  process  pi  will  know  how  to  compute  $  and  will  send  a  message 
to  Po  when  its  local  state  changes  in  a  way  significant  with  respect  to  $.  In 
particular,  a  process  can  determine  whether  a  local  event  potentially  changes 


More  formally,  let  $  be  a  predicate  expressed  over  a  global  state;  that 
is,  $(3i,. .  .,5n)  is  true  or  false.  Consider  some  event  e‘  of  process  p;;  recall 
that  is  the  value  of  5,  before  e\  executes  and  s-  is  the  value  of  .s,  after 
e-  executes.  Event  e-  potentially  affirms  $  if  the  execution  of  e-  could  have 
made  $  true: 

,  .  .  . ,  S|_x  ,  3|'.f  1 1  •  •  • ,  ,....  Sn  )  A  $(Si,.  .  ..S;,  .  . 

Similarly,  event  e'  potentially  rejects  $  if  the  execution  of  e-  could  have  made 
$  false: 

3si,...,S,_i,Si+i,...,Sn  :  $(Si,  .  .  - .S„)  A  . s' . Sr,). 

An  event  potentially  changes  $  if  it  potentially  affirms  or  rejects  such  an 
event  is  also  called  a  relevant  event. 

Note  that  an  event  can  both  potentially  affirm  and  reject  For  example, 
if  n  >  4  and  $  is  “either  two  or  three  processes  have  their  lights  on.’’  then 
when  a  process  turns  its  light  on,  this  action  both  potentially  affirms  and 
rejects  $  even  though  it  is  possible  that  the  value  of  $  did  not  change. 

Our  detection  protocols  will  have  the  monitored  processes  periodically 
send  to  the  monitor  its  state  relevant  to  that  is.  the  message  will  contain 
the  values  of  the  variables  of  pi  referred  to  in  $.  For  each  process  p,  ( 1  <  ;  < 
n),  process  po  maintains  a  sequence  Qi  of  such  messages  received  from  p,. 
These  messages  will  also  carry  information  for  ordering  these  states,  which 
is  described  next. 

3.1  Weak  Vector  Clocks  and  Enumeration  of  Global  States 

Our  protocols  will  have  the  monitor  enumerate  possible  global  states  of  the 
system  by  choosing  states  from  each  of  the  message  sequences  Q,.  In  this 
section,  we  describe  how  this  enumeration  of  global  states  is  performed.  We 
use  a  slight  modification  of  vector  clocks  [12]. 

A  logical  clock  [7]  is  a  value  T  that  satisfies  the  clock  condition:  given 
two  events  ei  and  ej  and  their  associated  clock  values  T{ex)  and  Tlei)-  if 
Cl  — *  eji  then  ^(®i)  <  T(e2).  We  will  find  it  advantageous  to  use  clocks  that 
also  satisfy  the  converse  of  the  clock  condition:  that  is.  clocks  that  satisfy 

{e\  -*  e2)  o  T{ei)  <  T[€2).  (1) 

In  particular,  such  clocks  enable  one  to  determine  whether  or  not  two  events 
are  concurrent;  ei  and  62  are  concurrent  if  neither  fi  —  €2  'tor  £2  —  fi- 
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Unfortunately,  Lamport’s  logical  clocks  of  [7]  {which  are  implemented  using 
a  single  counter)  do  not  satisfy  Equation  1. 

A  logical  clock  that  satisfies  Equation  1  can  be  implemented  with  a 
vector  U  of  n  counters.  If  Vj  is  the  logical  clock  associated  with  process  />,. 
then  Vi[i]  is  the  number  of  events  that  have  been  executed  by  p;  and  V’[j]. 
j  ^  i,  is  the  number  of  events-  that  p,  •‘knows”  pj  has  executed.  If  e,  is 
an  event  at  process  i,  then  we  use  V(ei)  to  denote  the  value  of  V'  after  e, 
executed.  Given  this  definition,  one  can  easily  show  that  —  Cj  if  and  only 
if  the  vector  clock  of  ej  records  the  fact  that  has  occurred: 

e,  ^  ej  V(ei){i]  <  V{ej)[i].  (2) 

Similarly,  if  and  ej  are  concurrent,  then 

U(e,)[i]  >  U(ej)[i]A  V^(ej)[j]  >  U(e,)[j]. 

If  the  set  of  processes  is  static,  then  vector  clocks  are  not  hard  to  im¬ 
plement.  Initially,  Vi[j]  is  set  to  zero  for  all  i  and  j.  Vi[j]  is  incremented 
whenever  pi  executes  an  event.  Every  message  sent  by  pi  is  tiniesiamped 
with  Vi  (let  U(m)  refer  to  the  timestamp  on  message  m).  If  e,  is  the  receipt 
of  message  m,  then  each  Vi[A:]  {k  ^  i)  is  set  to  the  maximum  of  \',[k]  and 
U(m)(fc].  As  an  example.  Figure  3  shows  the  values  of  vector  clocks  for  the 
events  of  the  execution  shown  in  Figure  2. 

We  can  use  vector  clocks  to  determine  whether  or  not  a  set  of  local 

states  represents  a  consistent  cut.  The  set  of  local  states  S  =  (si . s„)  is 

a  (consistent)  global  state  if  every  pair  of  local  states  s,  and  Sj  is  potentially 
concurrent.  In  terms  of  vector  clocks,  s,  and  Sj  are  potentially  concurrent 
if  U(s,)[i]  >  U(sj)[i]  and  U(sj)(j]  >  V'(5.)[j].  Thus,  the  global  state  S  is  a 
consistent  cut  if  and  only  if 

Vi,j  :  1  <  i,i  <  n  :  U(s,){/:]  >  V'(S;)[j].  (3) 

Because  we  are  interested  only  in  the  causal  relationship  of  events  that 
potentially  change  $,  we  can  use  a  slight  weakening  of  vector  clocks  [10]. 
With  our  clocks,  process  p,  will  increment  its  local  counter  U,[/]  only  when 
it  executes  an  event  that  potentially  changes  It  will  send  a  message 
to  po  whenever  its  vector  clock  changes — that  is.  either  when  it  e.xecutes  a 
relevant  event  or  when  it  e.xecutes  a  receive  event  through  which  it  learns 
that  another  process  has  potentially  changed  The  message  sent  from  p, 
to  Po  will  contain  p,’s  state  after  such  an  event  is  executed  and  the  vector 
time  V(si). 
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Figure  3:  Vector  Clocks  for  Events  of  Figure  2 


Figure  4  illustrates  such  vector  clocks.  These  clocks  are  a  weakened 
version  of  normal  vector  clocks — for  example,  if  i  =  j.  they  need  not  satisfy 
Equation  2.  They  do,  however,  satisfy  Equation  3.  and  this  is  all  that 
our  protocols  need.  For  the  sake  of  simplicity,  the  remainder  of  this  paper 
cissumes  that  all  events — including  send  and  receive  events —  are  relevant 
events  and  thus  that  our  weak  vector  clocks  are  true  vector  clocks. 


3.2  Asynchronous  Systems 

In  this  section,  we  assume  that  processes  do  not  possess  local  real-time 
clocks,  that  there  is  no  global  clock,  and  that  there  is  no  upper  bound  on 
message  delays.  We  note  in  advance  that  there  is  no  way  to  bound  the 
amount  of  time  between  the  time  a  condition  becomes  true  and  the  time  the 
monitor  detects  the  condition.  This  is  because  messages  sent  to  the  monitor 
may  be  arbitrarily  delayed. 

The  protocols  for  detecting  possibly  $  and  definitely  $  'le  based  on  the 
same  data  structure:  the  lattice  of  consistent  global  states  that  correspond 
to  an  observed  execution.  Such  a  lattice  consists  of  n  orthogonal  a.xes.  with 

one  axis  for  each  monitored  process.  A  point  f  =  (ti.  <2 . t„)  in  this  lattice 

corresponds  to  a  consistent  global  state  in  which  process  p,  has  executed  t, 
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(1,0,0)  (2.2.2) 


Figure  4:  Weak  Vector  Clocks  for  Events  of  Figure  2 


events.  Of  course,  not  all  tuples  ■  ■  -Jn)  appear  in  the  lattice:  this 

depends  on  the  causal  dependencies  among  the  local  states  of  P.  Define  the 
level  of  a  point  t  to  be  the  sum  of  its  indices  ti  +  t2  +  ■  ■  ■  +  in- 

Consider  some  global  history.  A  linearization  of  this  history  is  a  total 
order  of  (consistent)  global  states  in  which  exactly  one  process  executes  one 
event  between  adjacent  global  states  In  terms  of  the  lattice  corresponding 
to  the  history,  a  linearization  corresponds  to  a  path  in  the  lattice,  where  the 
level  of  each  subsequent  point  in  the  path  increases  by  one.  space-time 
diagram  of  a  two-process  system  and  the  corresponding  lattice  of  global 
states  is  illustrated  in  Figure  5.  A  point  5,j  represents  a  state  in  which 
process  pi  is  in  its  state  and  process  p2  is  in  its  state.  From  the  lattice, 
it  is  easy  to  see  that  one  possible  e.xecution  corresponds  to  the  sequence  of 
global  states 

■S'oo;  ■S'oi',  S’n;  S2\:  S22’  -Saa'.  5^3  •  ■  •  • 

For  every  point  7  in  a  lattice,  there  e.xists  at  least  one  linearization  that 
passes  through  t.  Hence,  if  any  point  in  the  lattice  satisfies  $.  then  po.^.cibly 
$  holds.  The  property  definitely^  requires  all  linearizations  to  pass  through 
a  point  that  satisfies  $.  For  e.xample,  suppose  in  Figure  5  that  the  points  .S'^3 
and  534  both  represent  states  that  satisfy  <&;  then  definitely  $  Iiolds.  This  is 
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because  543  and  534  are  the  only  points  in  level  7  and  all  linearizations  must 
pass  through  some  point  in  that  level.  Definitely  $  also  holds  if  instead  the 
states  represented  by  points  in  the  set  {553. 535. 554, 545}  all  satisfy  <P.  This 
is  because  if  a  linearization  does  not  pass  through  553  or  535.  then  it  must 
pass  through  544  and  hence  through  either  5.54  or  S45. 

Figures  6  and  7  give  the  high-level  algorithms  that  a  monitoring  process 
uses  to  detect  possibly  $  and  definitely  <&,  respectively,  in  an  n-processor 
system.  Each  algorithm  begins  by  having  the  monitor  distribute  the  predi¬ 
cate  $  to  all  processes  and  then  construct  the  initial  global  state  of  level  0. 
(It  is  assumed  that  the  monitor  knows  a  priori  each  process’s  initial  state 
relevant  to  if  this  is  not  the  case,  the  processes  begin  by  performing  a 
two-phase  synchronization  protocol.) 

The  possibly  $  algorithm  is  straightforward:  using  the  messages  it  re¬ 
ceives,  the  monitor  iteratively  constructs  levels  of  the  lattice,  using  the  vec¬ 
tor  timestamps  accompanying  each  message  (see  below).  If  it  ever  finds  a 
global  state  in  the  current  level  satisfying  $.  then  it  reports  possibly  <I>  and 
halts.  Note  that  this  protocol  is  not  optimal  in  its  reporting  time  because 
it  always  waits  for  a  level  to  be  completely  enumerated.  This  restriction  is 
not  necessary  and  is  only  done  to  simplify  the  presentation  of  the  algorithm 
and  the  one  that  follows. 

The  definitely  $  algorithm  also  iteratively  constructs  one  level  at  a  time. 
It  attempts  to  prove  that  all  paths  in  the  lattice  pass  through  a  state  sup¬ 
porting  $.  To  this  end,  when  constructing  a  new  level,  it  adds  only  states 
that  do  not  support  call  the  resulting  level  a  reduced  lerel.  If  the  monitor 
ever  finds  an  empty  reduced  level,  then  the  monitor  halts  and  reports  d( fi¬ 
nitely  $  (in  fact,  it  can  report  that  ^  definitely  holds  by  the  time  processes 
execute  a  total  of  Ivl  relevant  events,  where  Ivl  is  the  last  level  enumerated). 

As  stated  earlier,  the  implementations  of  the  algorithms  in  Figures  (> 
and  7  require  a  monitored  process  to  send  the  relevant  part  of  its  local  'tare 
to  the  monitor  whenever  its  vector  clock  changes.  The  monitor  maintains 
sequences  of  these  states,  one  per  process,  and  assembles  them  into  the 
necessary  global  states.  Thus,  the  monitor  must  be  able  to  determine  wiien 
it  can  assemble  all  the  reachable  global  states  of  a  given  level  and  when 
it  can  drop  a  local  state  from  its  sequence  because  the  local  state  cannot 
appear  in  any  further  global  states  of  interest.  To  achieve  this,  we  use  weak 
vector  timestamps  developed  in  the  previous  section. 

Let  Qi  be  the  sequence  of  messages  that  po  has  received  from  p,  'tored  in 
FIFO  order.  Each  state  s,  in  a  message  stored  in  Q,  is  labeled  with  the  weak 
vector  timestamp  F(s,)  of  the  event  that  generated  that  state.  Ecpiation  :) 
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defines  when  a  set  of  states  {si,S2, . .  with  s,  from  process  p,.  comprise 
a  consistent  cut.  Note  that  the  level  of  this  global  state  is  ^"=1  r(.Su)[u]. 

Consider  some  point  t  =  (<i, in  the  lattice  that  corresponds  to 
the  state  {si, . .  .,Sn)-  The  monitor  can  enumerate  points  of  the  next  level 
in  the  lattice  as  follows.  For  each  process  p,,  the  monitor  checks  to  see  if  s', 
the  state  in  the  (ti  +  1)^‘  message  of  Qi,  is  potentially  concurrent  with  the 
other  Sj's  (if  there  is  no  such  state  in  Q,,  the  monitor  cannot  complete  the 
next  level  until  it  receives  that  state).  Thus,  if 

'ij  :j  V(s')[i]  >  K(sj)(f]  A  C(.Sj)[j]  >  V''(s')[;]  (4) 

then  point  (t^, . . . ,  t,  +  1, . . . ,  t„)  is  in  the  lattice  at  the  next  level.  ( .Although 
many  such  states  will  have  be  checked,  it  should  be  clear  that  a  state  at  some 
level  in  the  lattice  may  follow  from  several  in  the  previous  level:  it  only  has 
to  be  checked  once  and  not  for  each  possible  predecessor.) 

We  can  also  use  vector  timestamps  to  determine  when  a  message  con¬ 
taining  state  Si  can  be  eliminated,  in  the  interest  of  saving  space,  from  a 
queue  Qi.  If  the  last  state  in  each  other  queue  happens  after  s,  and  is  not 
potentially  concurrent  with  it,  then  no  state  subsequently  received  could 
possibly  form  a  global  state  with  Sj.  Thus,  the  message  containing  s,  can 
be  removed  from  Qi  as  soon  as  the  following  holds: 

Vj  V(Qj.last)[i]  >  F(.s,)[/]. 

where  Qj.last  is  the  last  state  in  Qj. 

The  running  time  of  both  detection  algorithms  are  linear  in  the  number  of 
global  states.  Unfortunately,  the  number  of  global  states  can  be  exponential 
in  the  number  of  processes.  Even  worse,  the  worst-case  space  complexity 
is  unbounded,  since  the  delivery  of  a  message  can  be  indefinitely  delayed 
in  an  asynchronous  system.  While  there  are  heuristics  that  can  be  used  to 
limit  the  number  of  constructed  global  states,  they  are  intrusive  in  that  they 
require  some  kind  of  synchronization  or  limited  blocking  of  the  monitored 
processes.  Real-time  bounds  on  communication  and  the  rate  of  change  of 
local  states  can  also  be  used,  as  is  discussed  in  the  next  section. 

3.3  Partially  Synchronous  Systems 

In  this  section,  we  assume  that  each  process  p,  has  a  real-time  clock  C,.  and 
that  these  clocks  are  appro.ximately  synchronized:  at  any  given  "rear’  time, 
the  difference  between  the  clocks  of  two  processes  is  no  more  than  t.  We 
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define  this  formally  by  modifying  our  definition  of  histories  and  linearizations 
slightly.  Firstly,  all  processes  (including  po)  execute  "tick”  events;  a  process's 
local  time  is  the  number  of  tick  events  that  it  has  executed.  If  f ,  is  an  event 
at  Pi,  then  C,(ei)  is  the  number  of  tick  events  that  p,  has  executed  through 
e,.  If  /f  is  a  history  with  approximately  synchronized  clocks,  then  L  is  a 
linearization  of  H  only  if,  in  addition  to  the  usual  requirement,  in  all  prefixes 
of  L  and  every  pair  of  processes  p,  and  Pj,  the  difference  in  the  number  of 
tick  events  executed  by  the  two  processes  is  at  most  t. 

In  addition  to  approximately  synchronized  clocks,  we  assume  that  there 
are  lower  and  upper  bounds  on  message  transmission  times.  This  means  that 
if  process  p,-  e.xecutes  "send  m  to  pj"  after  it  has  executed  tg  tick  events,  then 
when  Pj  executes  "receive  m  from  p,.”  it  has  executed  tr  tick  events,  where 
+  dtnin  ^  +  dmax  for  Constants  d^in  and  d^ax  (both  greater  than 

0).  These  bounds  will  be  especially  important  when  considering  messages 
received  by  the  monitor  po.  Approximately  synchronized  clocks  can  be  used 
to  e.xtend  the  "happens  before”  relation  to  order  two  events  e,  and  Cj  even 
when  there  is  no  explicit  communication  between  p,  and  pj-.  thus,  we  redefine 
e,  —  ey. 


ei  (1/(6, )[2]  <  V'(ej)(t])  V  (C(e,)  +  f  <  C’(ej )).’ 

That  is,  ti  must  happen  before  Cj  either  if  e,  can  causally  affect  f.,  (as 
measured  by  vector  timestamps)  or  if  the  clock  times  corresponding  to  the 
events  show  that  e;  must  happen  first. 

Our  protocols  will  be  such  that  each  state  sent  by  a  process  p,  to  the 
monitoring  process  po  will  include  the  local  time  C(.s,)  at  which  the  event 
resulting  in  occurred,  as  well  as  the  vector  timestamp  f '( s, ).  The  monitor 
can  then  use  the  vector  timestamps  and  the  clock  times  of  these  states  to 
enumerate  the  levels  of  the  lattice.  The  clock  times  can  be  used  to  further 
restrict  the  pairs  of  events  that  are  potentially  concurrent.  With  each  state 
Si  in  Qi,  the  monitor  can  determine  the  latest  local  time  at  which  p,  must 
have  been  in  state  s,  (call  this  L(s,)).  If  there  is  a  state  s[  after  -s,  in  Q,. 
then  this  is  C(sJ);  if  s,  =  Q,.last,  then  this  is  C  -  dmax-  where  C  is  the 
monitor’s  current  local  time.  If  p,  had  changed  its  local  state  between  (('(  .'•, ) 
and  C  —  dniaxt  then  the  monitor  would  have  gotten  another  message  from  p, 
by  its  local  time  C. 

'There  is  no  need  to  take  the  transitive  closure  of  the  two  relations  because,  if  i/mm  >  b, 
C{e,)  +  e  <  C(ek)  then  C(e, )  +  r  <  T'lei ).  and  if  flc , )  +  f  <  C{f  ,] 
and  V(e,)[j]  <  t^(efc)[j)  then  C(e,)  +<  <  C(ei,). 
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We  can  now  say  that  two  states  a,  and  Sj  received  by  the  monitor  are 
potentially  concurrent  if  both  the  vector  time  stamps  and  the  the  real-time 
clocks  indicate  this: 

(l/(s.)[t]>  V^(5,)[j])  A  (V(s,)lj]>V(s,)[i]) 

A  ((C(si)  -  f)  <  C(Sj)  <  (5) 

V  (C(5_,)  -  f)  <  C(Si)  < 

Suppose  now  that  the  monitor  is  seeking  to  extend  the  state  (sj . s„)  to 

the  next  level  by  potentially  adding  a  new  state  s',  the  (f,  -I- 1)*'  state  in  Q,. 
It  checks  to  see  if  s'  is  potentially  concurrent  with  the  other  Sj's  by  using 
Equation  5  instead  of  Equation  4.  If  s'  is  potentially  concurrent  with  all 

the  Sj’s,  then  the  state  (si,...,s' . s„)  is  added  to  the  next  level  of  the 

lattice;  otherwise,  it  is  not.  An  e.xception  to  the  last  point  is  if  s'  = 
and  s'i  was  not  deemed  to  be  potentially  concurrent  because  its  latest  time 
was  too  early.  For  example,  suppose  e  =  1  and 

C(s')  =  3;  i(s-)  =  4;  C(Sj)  =  6:  Z,(Sj)  =  7. 

Because  sJ  =  Qi.last,  X(s')  —  C  -  dma-xi  as  time  passes  on  the  monitor's 
clock,  L(Si)  may  grow  so  that  the  two  states  would  be  judged  potentially 
concurrent.  In  such  cases,  therefore,  the  decision  about  whether  to  add 
the  state  (si, . .  .,s';, . .  ..Sn)  to  the  lattice  is  postponed  until  either  another 
message  arrives  from  p,  or  the  monitor's  clock  advances  to  a  point  where  a 
decision  can  be  made.  Until  then,  the  level  cannot  be  completely  enumer¬ 
ated. 

The  conditions  possibly  $  and  definitely  $  can  now  be  detected  exactly 
as  in  the  previous  section.  Each  processor  sends  its  state  to  the  monitor 
whenever  its  vector  clock  changes;  it  includes  with  this  message  its  vector 
time  and  the  number  of  tick  events  it  has  executed.  The  monitor  then  uses 
this  information  to  construct  levels  of  the  lattice,  using  the  properties  of  the 
“potentially  concurrent”  states  discussed  above.  It  then  reports  "possibly 
or  ‘^definitely  exactly  as  it  would  in  the  case  of  asynchronous  systems. 

We  now  argue  upper  bounds  on  detection  times.  Suppose  that  S  = 
(si,...,Sn)  is  a  global  state  such  that  the  last  event  leading  to  this  state 
occurs  when  the  monitor’s  local  time  is  i.  No  process's  local  clock  is  higher 
than  t  +  e  when  one  of  the  events  leading  to  5  occurs,  so  po  receives  all 
messages  necessary  to  construct  this  state  by  local  time  t  +  e  +  dmux-  Local 
time  t  -h  2e  is  the  latest  that  a  process  could  e.xecute  an  event  that  could  be 
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potentially  concurrent  with  one  leading  to  5;  thus,  by  time  t->r2e  +  f/max-  Po 
will  have  completed  the  construction  of  the  level  containing  5. 

Suppose  that  possibly  $  holds  of  a  history;  this  means  that  some  consis¬ 
tent  cut  of  the  history  supports  If  the  last  event  leading  to  this  cut  occur 
when  the  monitor’s  local  clock  is  t,  then  the  monitor  will  finish  enumerating 
the  level  of  5  at  its  local  time  t  +  2£  +  d,nax>  detecting  possibly  $  at  that  time 
(actually,  it  could  detect  it  at  time  t  +  e-i-dmax  because,  as  noted  earlier,  the 
possibly  protocol  does  not  need  to  enumerate  an  entire  level  once  it  finds  a 
state  satisfying  $). 

Suppose  that  definitely  $  holds  of  a  history:  this  means  that  there  is  some 
finite  set  of  consistent  cuts,  all  supporting  $.  through  which  all  linearizations 
pass.  If  the  last  event  leading  to  the  last  of  these  occurs  when  the  monitor  s 
local  time  is  t,  then  the  monitor  must  detect  definitely  by  time  t -r2(  +  dmA\ 
on  its  local  clock;  this  is  because  the  Itist  state  in  the  level  of  the  last  cut 
will  be  enumerated  by  that  time,  and  the  protocol  will  halt. 

The  above  discussion  does  not  consider  the  amount  of  local  computation 
required  by  the  monitor.  In  general,  this  depends  on  the  relation  between  c 
and  the  rate  at  which  processes  can  potentially  change  <^.  If  clocks  are  closely 
synchronized,  then  the  monitor  will  never  have  to  consider  more  than  a  few 
state  changes  by  any  one  process.  If  instead  processes  potentially  change  $ 
very  often,  then  the  monitor  may  have  to  do  significant  local  computation. 

4  Conclusions 

This  paper  has  defined  two  means  (possibly  and  definitely)  by  which  global 
states  in  an  asynchronous  or  loosely  synchronous  system  can  be  detected. 
It  presented  algorithms  by  which  a  monitor  can  detect  these  properties  in 
both  types  of  systems.  There  are  other  means  of  detection  that  are  also  of 
interest.  For  example,  we  have  been  investigating  a  third  type  of  detection, 
called  currently,  that  occurs  when  the  monitor  learns  a  condition  actually 
holds  at  the  time  of  detection.  One  can  modify  our  definitely  algorithms 
for  partially  synchronous  systems  to  detect  currently  b>  requiring  that  ap¬ 
plication  processes  forgo  potentially  rejecting  the  condition  being  detected 
for  a  well-defined  amount  of  time.  We  can  obtain  currently  algorithms  for 
asynchronous  systems  only  by  forcing  application  processes  to  block. 

These  algorithms  may  be  complex,  both  in  terms  of  computation  and 
storage.  Although  we  are  investigating  optimizations  of  these  algorithms, 
we  maintain  that  significant  comple.xity  is  required  for  detection  to  be  com¬ 
plete.  In  the  future,  we  plan  to  look  at  the  kinds  of  information  that  may 
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simplify  the  detection.  If  the  property  that  is  to  be  detected.  $.  has  cer¬ 
tain  nice  properties,  then  detection  may  be  simplified.  If  the  monitor  has 
some  knowledge  of  the  how  and  when  the  application  program  potentially 
changes  the  condition  to  be  detected,  then  this  can  also  simplify  detection. 
We  have  also  been  investigating  casting  the  detection  problem  into  temporal 
and  epistomological  logics.  We  believe  that  such  a  characterization  will  aid 
in  finding  sets  of  properties  under  which  detection  can  be  simplified. 

Although  our  original  application  was  towards  distributed  application 
management,  we  have  also  been  investigating  the  use  of  these  detection 
protcols  in  the  scope  of  debugging  distributed  systems  [-3].  The  constraints 
of  a  debugger  are  slightly  different  from  those  that  arise  in  tool  integration 
or  distributed  application  management.  For  e.xample.  invasiveness  is  tradi¬ 
tionally  considered  untolerable,  yet  in  tool  integration,  temporarily  blocking 
an  application  may  be  acceptable. 

The  work  most  similar  in  spirit  to  ours  are  tlie  protocols  developed  by 
Spezialetti  [16].  In  particular,  her  event  holding  condition  is  the  same  spec¬ 
ification  as  our  protocol  for  detecting  currently  <>.  and  the  specification  of 
her  event  occurrence  condition  is  similar  to  the  specification  of  our  po.^xibly 
$  algorithm.  However,  her  protocols  for  non-local  event  detection  are  in¬ 
complete,  in  that  they  can  miss  conditions  that  in  fact  held.  For  example, 
the  execution  in  Figure  8  shows  such  an  execution.  If  the  messages  in  this 
figure  correspond  to  the  messages  generated  in  establishing  simultaneous 
regions  [15],  then  her  protocol  will  not  detect  .ri  =  .t  j.  yet  in  fact  /lefin/teh/ 
Xi  =  i2  holds. 
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Figure  5:  An  execution  and  the  corresponding  lattice  of  global  states. 
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Possibly(<^}:  begin 

current  :=  global  state  {sj,  Sj, . . . ,  6°); 

Ivl  :=  0; 

do  no  state  in  current  satisfies 
last  :=  current 
Ivl  ;=  Ivl  +  1; 

current  :=  states  of  level  Ivl  reachable  from  a  state  in  last. 
od 
end: 

report  Possibly 

Figure  6:  Algorithm  for  detecting  Possibly 


Definitely(4):  begin 

last  :=  global  state  {s°,  sS, . . . ,  s®); 
remove  all  states  in  last  that  satisfy 
Ivl  :=  1; 

%  Invariant:  last  contains  all  states  of  level  It  I  —  1  that  are  accessible 
%  from  (si.Sj, . .  .,s°)  without  passing  through  a  state  satisfying  (l> 
do  last  ^  — 

current  :=  states  of  level  Ivl  reachable  from  a  state  in  last 
remove  all  states  in  current  that  satisfy 
Ivl  ;=  Ivl  +  1;  last  ;=  current 
od 
end: 

report  Definitely 

Figure  7:  Algorithm  for  detecting  Definitely 
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Xi  :=  4  Xi  :=  3 


Figure  8:  $  =  (xi  =  X2) 
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